Ipmicra: Toward a Distributed and Adaptable Location Aware Web Crawler

IPMicra: An IP-address based Location Aware Distributed Web Crawler

Distributed crawling is able to overcome important limitations of the traditional single-sourced web crawling systems. However, the optimal benefit of distributed crawling is usually limited to the sites hosting the crawlers, the rest of the URLs are by large randomly distributed to the various crawlers. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP addre...

Minimizing the Network Distance in Distributed Web Crawling

Distributed crawling has shown that it can overcome important limitations of the centralized crawling paradigm. However, the distributed nature of current distributed crawlers is currently not fully utilized. The optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work we describe IPMicra, a distributed location aware web crawler that utilizes an IP a...

On Location Aware Internet

An important aspect of performance for Internet-based applications is network delay (measured in terms of bandwidth and latency). The contribution and the basic motivation of Location Aware Internet is to enable clients to easily find the nearest (in terms of latency) out of a number of servers that can service a specific request. Location Aware Internet can significantly improve the performanc...

Location Aware Web-crawling with the Use of Migrating Crawlers

Focusing Web Crawls On Location-Specific Content

Retrieving relevant data for location-sensitive keyword queries is a challenging task that has so far been addressed as a problem of automatically determining the geographical orientation of web searches. Unfortunately, identifying localizable queries is not sufficient per se for performing successful location-sensitive searches, unless there exists a geo-referenced index of data sources agains...